Data converting apparatus, data converting method, learning apparatus, leaning method, program, and recording medium

ABSTRACT

The present invention relates to a data conversion apparatus and a learning device in which an image can be converted into a higher-quality image. A class-tap generating circuit ( 2 ) and a predictive-tap generating circuit ( 3 ) generate, from an SD image, a predictive tap used for determining the HD pixel of a specified block of an HD image. A classification circuit ( 4 ) classifies the HD pixel of the specified block based on the class tap. A coefficient RAM ( 7 ) obtains a tap coefficient for the class of the HD pixel of the specified block from tap coefficients obtained by learning the relationship between supervisor data and learner data for each class of at least one class by giving a predetermined constraint condition to the supervisor data. A predictive-computation circuit ( 8 ) and a decoding circuit ( 9 ) determine the HD pixel of the specified block by using the tap coefficient and the predictive tap.

TECHNICAL FIELD

The present invention relates to data conversion apparatuses andmethods, learning devices and methods, programs, and recording media.More particularly, the invention relates to a data conversion apparatusand method, a learning device and method, a program, and a recordingmedium in which image data, for example, can be converted intohigher-quality image data.

BACKGROUND ART

The applicant of this application previously proposed classificationadaptive processing as data conversion processing for improving thequality of images or performing other types of image conversion.

The classification adaptive processing includes classificationprocessing and adaptive processing: data is classified by classificationprocessing according to the property of the data, and each class of thedata is subjected to adaptive processing. The adaptive processing is,for example, as follows.

In the adaptive processing, for example, low-quality or standard-qualityimage (hereinafter sometimes referred to as an “SD (Standard Definition)image”) data is mapped by using predetermined tap coefficients so as tobe converted into high-quality image (hereinafter sometimes referred toas an “HD (High Definition) image”) data.

It is now assumed that, for example, a linear coupling model is employedas the mapping method using tap coefficients. In this case, the pixelvalues of pixels y forming HD image data (hereinafter sometimes referredto as “HD pixels”) are determined by using tap coefficients and aplurality of pixels forming SD image data (hereinafter sometimesreferred to as “SD pixels”) extracted as predictive taps for predictingthe HD pixels according to the following linear equation (linearcoupling). $\begin{matrix}{y = {\sum\limits_{n = 1}^{N}{w_{n}x_{n}}}} & (1)\end{matrix}$

In equation (1), x_(n) indicates the pixel value of the n-th pixel ofthe SD image data forming the predictive taps for the HD pixel y, andw_(n) indicates the n-th tap coefficient to be multiplied with the pixelvalue of the n-th SD pixel. In equation (1), it is assumed that thepredictive taps consist of N SD pixels x₁, x₂, . . . , x_(N).

The pixel value y of the HD pixel may be determined by equations ofhigher degrees, such as a quadratic equation, rather than by the linearequation expressed in (1).

When the true value of the pixel value of the k-th sample HD pixel isindicated by y_(k), and when the predictive value of the true valuey_(k) determined by equation (1) is indicated by y_(k)′, the predictiveerror e_(k) is expressed by the following equation.e _(k) =y _(k) −y _(k)′  (2)

Since the predictive value y_(k)′ in equation (2) is determined byequation (1), equation (1) is substituted into y_(k)′ in equation (2),thereby obtaining the following equation. $\begin{matrix}{e_{k} = {y_{k} - \left( {\sum\limits_{n = 1}^{N}{w_{n}x_{n,k}}} \right)}} & (3)\end{matrix}$

In equation (3), x_(n,k) designates the n-th SD pixel forming thepredictive taps for the k-th sample HD pixel.

The tap coefficient w_(n) that sets the predictive error e_(k) to be 0in equation (3) is the optimal value for predicting the HD pixel.Generally, however, it is difficult to determine such tap coefficientsw_(n) for all the HD pixels.

Accordingly, as the standard for the optimal tap coefficient w_(n), themethod of least squares, for example, is used. Then, the optimal tapcoefficient w_(n) can be determined by minimizing the sum E of thesquare errors as the statistical error expressed by the followingequation. $\begin{matrix}{E = {\sum\limits_{k = 1}^{K}e_{k}^{2}}} & (4)\end{matrix}$

In equation (4), K indicates the number of set samples of the HD pixely_(k) and the SD pixels x_(1,k), x_(2,k), . . . , x_(N,k) forming thepredictive taps for the HD pixel y_(k).

The tap coefficient w_(n) that minimizes the sum E of the square errorsin equation (4) must satisfy the condition that the value determined bypartial-differentiating the sum E with the tap coefficient w_(n) becomes0, and thus, the following equation must be established. $\begin{matrix}{\frac{\partial E}{\partial w_{n}} = {{{e_{1}\frac{\partial e_{1}}{\partial w_{n}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{n}}} + \ldots + {e_{k}\frac{\partial e_{n}}{\partial w_{n}}}} = {0\quad\left( {{n = 1},2,\ldots\quad,N} \right)}}} & (5)\end{matrix}$

Accordingly, by partial-differentiating equation (3) with the tapcoefficient w_(n), the following equation can be found. $\begin{matrix}{{\frac{\partial e_{k}}{\partial w_{1}} = {- x_{1,k}}},{\frac{\partial e_{k}}{\partial w_{2}} = {- x_{2,k}}},\ldots\quad,{\frac{\partial e_{k}}{\partial w_{N}} = {- x_{N,k}}},\left( {{k = 1},2,\ldots\quad,K} \right)} & (6)\end{matrix}$

The following equation can be found from equations (5) and (6).$\begin{matrix}{{{\sum\limits_{k = 1}^{K}{e_{k}x_{1,k}}} = 0},{{\sum\limits_{k = 1}^{K}{e_{k}x_{2,k}}} = 0},{{\ldots\quad{\sum\limits_{k = 1}^{K}{e_{k}x_{N,k}}}} = 0}} & (7)\end{matrix}$

By substituting equation (3) into e_(k) in equation (7), equation (7)can be expressed by the normal equations expressed by equations (8).$\begin{matrix}{{\begin{bmatrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{N,k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{N,k}}} \right) \\\vdots & \vdots & ⋰ & \vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{N,k}}} \right)\end{bmatrix}\left\lbrack \quad\begin{matrix}w_{1} \\w_{2} \\\vdots \\w_{N}\end{matrix} \right\rbrack}\quad{\begin{matrix} = \\ = \\ = \end{matrix}\left\lbrack \quad\begin{matrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}y_{k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}y_{k}}} \right) \\\vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}y_{k}}} \right)\end{matrix}\quad \right\rbrack}} & (8)\end{matrix}$

By preparing a certain number of sets of the HD pixels y_(k) and the SDpixels x_(n,k), the same number of normal equations (8) as the number oftap coefficients w_(n) to be determined can be found, and by solvingequations (8) (the matrix at the left side next to the tap coefficientsw_(n) in equations (8) must be regular to solve equations (8), theoptimal tap coefficients w_(n) can be determined. In solving equations(8), the sweep-out method (Gauss-Jordan elimination), for example, maybe employed.

As described above, by solving equations (8) by setting many HD pixelsy₁, y₂, . . . , y_(K) to be supervisor data as supervisors for learningtap coefficients and by setting SD pixels x_(1,k), x_(2,k), . . . ,x_(N,k) forming the predictive taps for each HD pixel y_(k) to belearner data as learners for learning the tap coefficients, learning isconducted for determining the optimal tap coefficients w_(n). By usingthe optimal tap coefficients w_(n), SD image data is mapped (converted)onto (into) HD image data by using equation (1). The above-describedprocessing is adaptive processing.

The adaptive processing is different from mere interpolation processingin that components contained not in SD images but in HD images arereproduced. More specifically, only from equation (1), the adaptiveprocessing is similar to the so-called “interpolation processing” usinginterpolation filters. However, the tap coefficients w_(n), whichcorrespond to tap coefficients used in the interpolation filters, aredetermined by learning by using HD image data as supervisor data and SDimage data as learner data. Thus, components contained in HD images canbe reproduced. Accordingly, it is possible that the adaptive processingserves the function of creating images (creating the resolution).

In learning tap coefficients w_(n), the combinations of supervisor datay and learner data x can be changed so as to obtain tap coefficientsw_(n) performing various conversions.

If HD image data is used as the supervisor data y, and if SD image datadetermined by adding noise or blurring to the HD image data is used asthe learner data x, tap coefficients w_(n) for converting an image intoan image without noise or blurring can be obtained. If HD image data isused as the supervisor data y, and if SD image data determined bydecreasing the resolution of the HD image data is used as the learnerdata x, tap coefficients w_(n) for converting an image into an imagehaving improved resolution can be obtained. If image data is used as thesupervisor data y, and if DCT (Discrete Cosine Transform) coefficientsdetermined by performing DCT on the image data is used as the learnerdata x, tap coefficients w_(n) for converting the DCT coefficients intoimage data can be obtained.

As described above, in classification adaptive processing, the tapcoefficient w_(n) that minimizes the sum E of the square errors inequation (4) is determined for each class, and equation (1) iscalculated by using the tap coefficient w_(n), thereby converting an SDimage into a high-quality HD image. That is, by using the tapcoefficients w_(n) and the predictive taps x_(n) generated by the SDimage, equation (1) is calculated so as to determine HD pixels formingthe HD image.

Accordingly, in the classification adaptive processing previouslyproposed, when focusing on each HD pixel, the predictive value thatstatistically minimizes the predictive error of each HD pixel withrespect to the true value can be determined.

More specifically, it is now assumed, as shown in FIG. 1A, that thereare two HD pixels y_(k) and y_(k+1), which are horizontally, vertically,or obliquely adjacent to each other. In this case, as for the HD pixely_(k), the predictive value y_(k)′ that can statistically minimize thepredictive error e_(k) with respect to the true value y_(k) can beobtained. Similarly, as for the HD pixel y_(k+1) the predictive valuey_(k+1)′ that can statistically minimize the predictive error e_(k+1)with respect to the true value y_(k+1) can be obtained.

In the classification adaptive processing previously proposed, however,for the two HD pixels y_(k) and y_(k+1), when the true HD pixel y_(k)obliquely increases to the right side toward the true HD pixel y_(k+1),as those shown in FIG. 1A, the following result may be sometimes broughtabout. As shown in FIG. 1B, as for the HD pixel y_(k), the predictivevalue y_(k)′, which is greater than the true value, is obtained, and onthe other hand, as for the HD pixel y_(k+1) the predictive valuey_(k+1)′, which is smaller than the true value, is obtained.

In this case, the predictive value y_(k)′ for the HD pixel y_(k)decreases to the right side toward the predictive value y_(k+1)′ for theHD pixel y_(k+1), as shown in FIG. 1B.

As discussed above, if a predictive value decreases to the right side inspite of the fact that a true value increases to the right side, i.e.,if a change in the pixel value becomes opposite to a change in the truevalue, the resulting image quality may be decreased.

DISCLOSURE OF INVENTION

Accordingly, in view of this background, it is an object of the presentinvention to convert, for example, image data, into higher-quality imagedata.

A first data conversion apparatus of the present invention includes:class-tap generating means for generating, from first data, a class tapused for classifying a specified sample of second data into a class ofat least one class; classification means for classifying the specifiedsample based on the class tap; predictive-tap generating means forgenerating, from the first data, a predictive tap for determining thespecified sample; tap-coefficient obtaining means for obtaining a tapcoefficient for the class of the specified sample from tap coefficientsobtained by learning the relationship between supervisor datacorresponding to the second data, which serves as a learning supervisor,and learner data corresponding to the first data, which serves as alearner, for each of at least one class by giving a predeterminedconstraint condition to the supervisor data; and computation means fordetermining the specified sample by using the predictive tap and the tapcoefficient for the class of the specified sample.

A first data conversion method of the present invention includes: aclass-tap generating step of generating, from first data, a class tapused for classifying a specified sample of second data into a class ofat least one class; a classification step of classifying the specifiedsample based on the class tap; a predictive-tap generating step ofgenerating, from the first data, a predictive tap for determining thespecified sample; a tap-coefficient obtaining step of obtaining a tapcoefficient for the class of the specified sample from tap coefficientsobtained by learning the relationship between supervisor datacorresponding to the second data, which serves as a learning supervisor,and learner data corresponding to the first data, which serves as alearner, for each of at least one class by giving a predeterminedconstraint condition to the supervisor data; and a computation step ofdetermining the specified sample by using the predictive tap and the tapcoefficient for the class of the specified sample.

A first program of the present invention includes: a class-tapgenerating step of generating, from first data, a class tap used forclassifying a specified sample of second data into a class of at leastone class; a classification step of classifying the specified samplebased on the class tap; a predictive-tap generating step of generating,from the first data, a predictive tap for determining the specifiedsample; a tap-coefficient obtaining step of obtaining a tap coefficientfor the class of the specified sample from tap coefficients obtained bylearning the relationship between supervisor data corresponding to thesecond data, which serves as a learning supervisor, and learner datacorresponding to the first data, which serves as a learner, for each ofat least one class by giving a predetermined constraint condition to thesupervisor data; and a computation step of determining the specifiedsample by using the predictive tap and the tap coefficient for the classof the specified sample.

A first recording medium of the present invention records a programtherein, the program including: a class-tap generating step ofgenerating, from first data, a class tap used for classifying aspecified sample of second data into a class of at least one class; aclassification step of classifying the specified sample based on theclass tap; a predictive-tap generating step of generating, from thefirst data, a predictive tap for determining the specified sample; atap-coefficient obtaining step of obtaining a tap coefficient for theclass of the specified sample from tap coefficients obtained by learningthe relationship between supervisor data corresponding to the seconddata, which serves as a learning supervisor, and learner datacorresponding to the first data, which serves as a learner, for each ofat least one class by giving a predetermined constraint condition to thesupervisor data; and a computation step of determining the specifiedsample by using the predictive tap and the tap coefficient for the classof the specified sample.

A first learning device of the present invention includes: class-tapgenerating means for generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; classification means for classifying the specifieditem of data based on the class tap; predictive-tap generating means forgenerating a predictive tap used for determining the specified item ofdata from the learner data; and learning means for determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between thesupervisor data and the learner data for each of at least one class bygiving a predetermined constraint condition to the supervisor data.

A first learning method of the present invention includes: a class-tapgenerating step of generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; a classification step of classifying the specifieditem of data based on the class tap; a predictive-tap generating step ofgenerating a predictive tap used for determining the specified item ofdata from the learner data; and a learning step of determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between thesupervisor data and the learner data for each of at least one class bygiving a predetermined constraint condition to the supervisor data.

A second program of the present invention includes: a class-tapgenerating step of generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; a classification step of classifying the specifieditem of data based on the class tap; a predictive-tap generating step ofgenerating a predictive tap used for determining the specified item ofdata from the learner data; and a learning step of determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between thesupervisor data and the learner data for each of at least one class bygiving a predetermined constraint condition to the supervisor data.

A second recording medium of the present invention records a programtherein, the program including: a class-tap generating step ofgenerating a class tap used for classifying a specified item ofsupervisor data corresponding to second data, which serves as asupervisor for learning a tap coefficient, into a class of at least oneclass, from learner data corresponding to first data, which serves as alearner; a classification step of classifying the specified item of databased on the class tap; a predictive-tap generating step of generating apredictive tap used for determining the specified item of data from thelearner data; and a learning step of determining, by using the specifieditem of data and the predictive tap, the tap coefficient for each of atleast one class by learning the relationship between the supervisor dataand the learner data for each of at least one class by giving apredetermined constraint condition to the supervisor data.

A second data conversion apparatus of the present invention includes:class-tap generating means for generating, from first data, a class tapused for classifying a specified sample of second data into a class ofat least one class; classification means for classifying the specifiedsample based on the class tap; predictive-tap generating means forgenerating, from the first data, a predictive tap for determining thespecified sample; tap-coefficient obtaining means for obtaining a tapcoefficient for the class of the specified sample from tap coefficientsobtained by learning the relationship between a feature obtained from aplurality of samples of supervisor data corresponding to the seconddata, which serves as a learning supervisor, and a plurality of samplesof learner data corresponding to the first data, which serves as alearner, for each of at least one class; and computation means fordetermining the specified sample by using the predictive tap and the tapcoefficient for the class of the specified sample.

A second data conversion method of the present invention includes: aclass-tap generating step of generating, from first data, a class tapused for classifying a specified sample of second data into a class ofat least one class; a classification step of classifying the specifiedsample based on the class tap; a predictive-tap generating step ofgenerating, from the first data, a predictive tap for determining thespecified sample; a tap-coefficient obtaining step of obtaining a tapcoefficient for the class of the specified sample from tap coefficientsobtained by learning the relationship between a feature obtained from aplurality of samples of supervisor data corresponding to the seconddata, which serves as a learning supervisor, and a plurality of samplesof learner data corresponding to the first data, which serves as alearner, for each of at least one class; and a computation step ofdetermining the specified sample by using the predictive tap and the tapcoefficient for the class of the specified sample.

A third program of the present invention includes: a class-tapgenerating step of generating, from first data, a class tap used forclassifying a specified sample of second data into a class of at leastone class; a classification step of classifying the specified samplebased on the class tap; a predictive-tap generating step of generating,from the first data, a predictive tap for determining the specifiedsample; a tap-coefficient obtaining step of obtaining a tap coefficientfor the class of the specified sample from tap coefficients obtained bylearning the relationship between a feature obtained from a plurality ofsamples of supervisor data corresponding to the second data, whichserves as a learning supervisor, and a plurality of samples of learnerdata corresponding to the first data, which serves as a learner, foreach of at least one class; and a computation step of determining thespecified sample by using the predictive tap and the tap coefficient forthe class of the specified sample.

A third recording medium of the present invention records a programtherein, the program including: a class-tap generating step ofgenerating, from first data, a class tap used for classifying aspecified sample of second data into a class of at least one class; aclassification step of classifying the specified sample based on theclass tap; a predictive-tap generating step of generating, from thefirst data, a predictive tap for determining the specified sample; atap-coefficient obtaining step of obtaining a tap coefficient for theclass of the specified sample from tap coefficients obtained by learningthe relationship between a feature obtained from a plurality of samplesof supervisor data corresponding to the second data, which serves as alearning supervisor, and a plurality of samples of learner datacorresponding to the first data, which serves as a learner, for each ofat least one class; and a computation step of determining the specifiedsample by using the predictive tap and the tap coefficient for the classof the specified sample.

A second learning device of the present invention includes: class-tapgenerating means for generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; classification means for classifying the specifieditem of data based on the class tap; predictive-tap generating means forgenerating a predictive tap used for determining the specified item ofdata from the learner data; and learning means for determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between afeature obtained from a plurality of samples of the supervisor data anda plurality of samples of the learner data for each of at least oneclass.

A second learning method of the present invention includes: a class-tapgenerating step of generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; a classification step of classifying the specifieditem of data based on the class tap; a predictive-tap generating step ofgenerating a predictive tap used for determining the specified item ofdata from the learner data; and a learning step of determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between afeature obtained from a plurality of samples of the supervisor data anda plurality of samples of the learner data for each of at least oneclass.

A fourth program of the present invention includes: a class-tapgenerating step of generating a class tap used for classifying aspecified item of supervisor data corresponding to second data, whichserves as a supervisor for learning a tap coefficient, into a class ofat least one class, from learner data corresponding to first data, whichserves as a learner; a classification step of classifying the specifieditem of data based on the class tap; a predictive-tap generating step ofgenerating a predictive tap used for determining the specified item ofdata from the learner data; and a learning step of determining, by usingthe specified item of data and the predictive tap, the tap coefficientfor each of at least one class by learning the relationship between afeature obtained from a plurality of samples of the supervisor data anda plurality of samples of the learner data for each of at least oneclass.

A fourth recording medium of the present invention records a programtherein, the program including: a class-tap generating step ofgenerating a class tap used for classifying a specified item ofsupervisor data corresponding to second data, which serves as asupervisor for learning a tap coefficient, into a class of at least oneclass, from learner data corresponding to first data, which serves as alearner; a classification step of classifying the specified item of databased on the class tap; a predictive-tap generating step of generating apredictive tap used for determining the specified item of data from thelearner data; and a learning step of determining, by using the specifieditem of data and the predictive tap, the tap coefficient for each of atleast one class by learning a relationship between a feature obtainedfrom a plurality of samples of the supervisor data and a plurality ofsamples of the learner data for each of at least one class.

According to the first data conversion apparatus, the first dataconversion method, the first program, and the first recording medium ofthe present invention, a class tap used for classifying a specifiedsample of second data into a class of at least one class, and apredictive tap for determining the specified sample are generated fromfirst data. The specified sample is classified based on the class tap. Atap coefficient for the class of the specified sample is obtained fromtap coefficients obtained by learning the relationship betweensupervisor data corresponding to the second data, which serves as alearning supervisor, and learner data corresponding to the first data,which serves as a learner, for each of at least one class by giving apredetermined constraint condition to the supervisor data. The specifiedsample is determined by using the predictive tap and the tapcoefficient.

According to the first learning device, the first learning method, thesecond program, and the second recording medium of the presentinvention, a class tap used for classifying a specified item ofsupervisor data corresponding to second data, which serves as asupervisor for learning a tap coefficient, into a class of at least oneclass, and a predictive tap used for determining the specified item ofdata are generated from learner data corresponding to first data, whichserves as a learner, and the specified item of data is classified basedon the class tap. By using the specified item of data and the predictivetap, the tap coefficient for each of at least one class is determined bylearning the relationship between the supervisor data and the learnerdata for each of at least one class by giving a predetermined constraintcondition to the supervisor data.

According to the second data conversion apparatus, the second dataconversion method, the third program, and the third recording medium ofthe present invention, a class tap used for classifying a specifiedsample of second data into a class of at least one class, and apredictive tap for determining the specified sample are generated fromfirst data. The specified sample is classified based on the class tap. Atap coefficient for the class of the specified sample is obtained fromtap coefficients obtained by learning the relationship between a featureobtained from a plurality of samples of supervisor data corresponding tothe second data, which serves as a learning supervisor, and a pluralityof samples of learner data corresponding to the first data, which servesas a learner, for each of at least one class. The specified sample isdetermined by using the predictive tap and the tap coefficient for theclass of the specified sample.

According to the second learning device, the second learning method, thefourth program, and the fourth recording medium of the presentinvention, a class tap used for classifying a specified item ofsupervisor data corresponding to second data, which serves as asupervisor for learning a tap coefficient, into a class of at least oneclass, and a predictive tap used for determining the specified item ofdata are generated from learner data corresponding to first data, whichserves as a learner. The specified item of data is classified based onthe class tap. By using the specified item of data and the predictivetap, the tap coefficient for each of at least one class is determined bylearning the relationship between a feature obtained from a plurality ofsamples of the supervisor data and a plurality of samples of the learnerdata for each of at least one class.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a deterioration in the image quality.

FIG. 1B illustrates a deterioration in the image quality.

FIG. 2 is a block diagram illustrating an example of the configurationof a first embodiment of an image processing apparatus to which thepresent invention is applied.

FIG. 3 illustrates a block of HD image data, a class tap, and apredictive tap.

FIG. 4 is a flowchart illustrating processing performed by the imageprocessing apparatus.

FIG. 5 is a block diagram illustrating an example of the configurationof an embodiment of a learning device for learning tap coefficients tobe stored in a coefficient RAM 5.

FIG. 6 is a block diagram illustrating an example of the configurationof a learning-pair generating circuit 22.

FIG. 7 is a flowchart illustrating learning processing for learning tapcoefficients to be stored in the coefficient RAM 5.

FIG. 8 is a block diagram illustrating an example of the configurationof an embodiment of a learning device for learning tap coefficients tobe stored in a coefficient RAM 7.

FIG. 9 is a block diagram illustrating an example of the configurationof a learning-pair generating circuit 42.

FIG. 10 is a flowchart illustrating learning processing for learning tapcoefficients to be stored in the coefficient RAM 7.

FIG. 11 is a block diagram illustrating another example of theconfiguration of the learning-pair generating circuit 42.

FIG. 12 is a block diagram illustrating an example of the configurationof a second embodiment of the image processing apparatus to which thepresent invention is applied.

FIG. 13 illustrates another example of a block of HD image data.

FIG. 14 is a block diagram illustrating an example of the configurationof an embodiment of a computer to which the present invention isapplied.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 2 illustrates an example of the configuration of an embodiment ofan image processing apparatus to which the present invention is applied.

In this image processing apparatus, for example, a blurred SD image isinput, and classification adaptive processing is performed on theblurred SD image, thereby outputting an HD image in which blurring issufficiently reduced regardless of the level of blurring of the originalSD image.

More specifically, the image processing apparatus includes a framememory 1, a class-tap generating circuit 2, a predictive-tap generatingcircuit 3, a classification circuit 4, a frame memory 10, a main-pixelprocessing circuit 11, and a sub-pixel processing circuit 12. A blurredSD image is input into this image processing apparatus.

The frame memory 1 temporarily stores the SD image input into the imageprocessing apparatus in units of, for example, frames (or fields). Inthis embodiment, the frame memory 1 is able to store an SD imageconsisting of a plurality of frames by performing bank switching,thereby making it possible to perform image processing in real time evenif the SD image input into the image processing apparatus is a movingpicture.

The class-tap generating circuit 2 forms (divides) an HD image (ideal HDimage from which blurring is completely eliminated from the SD image) tobe determined by the classification adaptive processing into blocks,each block consisting of a plurality of HD pixels (samples), andsequentially specifies each block. The class-tap generating circuit 2then generates a class tap used for classifying HD pixels forming thespecified block from the SD image stored in the frame memory 1, therebyoutputting the generated class tap to the classification circuit 4. Thatis, the class-tap generating circuit 2 extracts, for example, aplurality of SD pixels (samples) positioned spatially or temporallyclose to the specified block, from the SD image stored in the framememory 1, and sets the extracted SD pixels as the class tap, therebyoutputting the class tap to the classification circuit 4.

More specifically, the class-tap generating circuit 2 divides, forexample, as shown in FIG. 3, an HD image into blocks, each blockconsisting of two vertically adjacent HD pixels, and sequentiallyspecifies each block.

In FIG. 3, ◯ represents SD pixels forming the SD image, and x designatesHD pixels forming the HD image. In FIG. 3, the number of pixels of theHD image in the vertical and horizontal directions is twice as that ofthe SD image.

The class-tap generating circuit 2 extracts, as shown in FIG. 3,three-row×three-column SD pixels which are vertically and horizontallyclose to the specified block from the SD pixels stored in the framememory 1, and sets the extracted SD pixels as a class tap.

The upper HD pixel and the lower HD pixel of a block divided from an HDimage are indicated by “y⁽¹⁾” and “y⁽²⁾”, respectively. Among 3×3 SDpixels forming a class tap, the first-row, the first-column SD pixel,the first-row, second-column SD pixel, the first-row, third-column SDpixel, the second-row, first-column SD pixel, the second-row,second-column SD pixel, the second-row, third-column SD pixel, thethird-row, first-column SD pixel, the third-row, second-column SD pixel,and the third-row, third-column SD pixel are indicated by x⁽¹⁾, x⁽²⁾,x⁽³⁾, x⁽⁴⁾, x⁽⁵⁾, x⁽⁶⁾, x⁽⁷⁾, x⁽⁸⁾, and x⁽⁹⁾, respectively.

The predictive-tap generating circuit 3 generates, from the SD imagestored in the frame memory 1, a predictive tap to be used fordetermining the predictive value of the HD pixel forming the specifiedblock in a predictive-computation circuit 6, and supplies the generatedpredictive tap to the predictive-computation circuit 6. That is, theclass-tap generating circuit 2 extracts, for example, a plurality of SDpixels positioned spatially or temporally close to the specified block,from the SD image stored in the frame memory 1, and sets the extractedSD pixels as the predictive tap, thereby outputting the predictive tapto the predictive-computation circuit 6.

For the sake of simplicity, for HD pixels forming a specified block, thepredictive-tap generating circuit 3 generates, a predictive tap havingthe same tap structure as that of the class tap, i.e., a predictive tapconsisting of 3×3 SD pixels.

However, it is not essential that the SD pixels serving as thepredictive tap and the SD pixels serving as the class tap are the same.That is, the predictive tap and the class tap may be independentlyformed (generated). The tap structure of the class tap or the predictivetap is not restricted to 3×3 SD pixels shown in FIG. 3.

Referring back to FIG. 2, the classification circuit 4 classifies the HDpixels forming the specified block based on the class tap output fromthe class-tap generating circuit 2, and supplies the class codecorresponding to the class of the HD pixels forming the specified blockto the main-pixel processing circuit 11 and the sub-pixel processingcircuit 12. That is, the classification circuit 4 performs one-bit ADRC(Adaptive Dynamic Range Coding) processing on the class tap output fromthe class-tap generating circuit 2, and outputs the resulting ADRC codeto the main-pixel processing circuit 11 and the sub-pixel processingcircuit 12 as the class code.

In K-bit ADRC processing, the maximum value MAX and the minimum valueMIN of the pixel values of the SD pixels forming the class tap aredetected, and DR=MAX−MIN is set to be the local dynamic range. Then,based on this dynamic range DR, the SD pixels forming the class tap arere-quantized into K bits. That is, the minimum value MIN is subtractedfrom the pixel value of each SD pixel forming the class tap, and theresulting value is divided (quantized) by DR/2^(K). Accordingly, whenperforming one-bit ADRC processing on the class tap, the pixel value ofeach SD pixel forming the class tap is re-quantized into one bit. Inthis case, the one-bit pixel values of the individual SD pixels formingthe class tap are arranged into a bit stream in a predetermined order,and the bit stream is output as the ADRC code. The classificationprocessing may be performed by another technique, for example, the SDpixels forming the class tap may be considered as vector components, andthe vectors may be quantized. In the classification processing, only oneclass may be used, in which case, the classification circuit 4 outputsfixed class code regardless of which class tap is supplied.

In this embodiment, in the class-tap generating circuit 2, the sameclass tap is generated for the HD pixels y⁽¹⁾ and y⁽²⁾ forming thespecified block. Accordingly, in the classification circuit 4, the HDpixels y⁽¹⁾ and y⁽²⁾ forming the specified block are classified into thesame class. Thus, in other words, in the classification circuit 4, theHD pixels forming the specified block are classified, and also thespecified block is classified.

The class-tap generating circuit 2 may generate class taps havingdifferent tap structures for the HD pixels y⁽¹⁾ and y⁽²⁾ forming thespecified block. Similarly, the predictive-tap generating circuit 3 mayalso generate predictive taps having different tap structures for the HDpixels y⁽¹⁾ and y⁽²⁾ forming the specified block. If, however, classtaps or predictive taps having different tap structures are generatedfor the HD pixels y⁽¹⁾ and y⁽²⁾ forming the specified block, it isnecessary that class codes or predictive taps determined from the classtaps generated for the HD pixels to be found in the main-pixelprocessing circuit 11 and the sub-pixel processing circuit 12 besupplied to the main-pixel processing circuit 11 and the sub-pixelprocessing circuit 12.

The frame memory 10 temporarily stores the HD pixels determined in themain-pixel processing circuit 11 and the HD pixels determined in thesub-pixel processing circuit 12, and when the HD pixels, for example,for one frame, are stored, one frame of an HD image consisting of the HDpixels is output. The frame memory 10 is configured similarly to theframe memory 1, and thus, it is able to store the HD pixels suppliedfrom the main-pixel processing circuit 11 and the sub-pixel processingcircuit 12 and read the HD pixels from the frame memory 10 at the sametime.

The main-pixel processing circuit 11, which is formed of a coefficientRAM (Random Access Memory) 5 and the predictive-computation circuit 6,specifies the main pixel of the HD pixels forming the specified block,determines the predictive value of the main pixel, supplies it to theframe memory 10, and stores it at the address corresponding to theposition of the main pixel.

The coefficient RAM 5 stores tap coefficients obtained by learning therelationship between supervisor data, which is HD pixel data serving asa learning supervisor, and learner data, which is SD image data servingas a learner, for each of at least one class. Upon receiving the classcode of the HD pixels of the specified block from the classificationcircuit 4, the coefficient RAM 5 reads the tap coefficient stored at theaddress corresponding to the class code so as to obtain the tapcoefficient of the class of the main pixel of the HD pixels forming thespecified block, and supplies the tap coefficient to thepredictive-computation circuit 6. Details of the learning method for tapcoefficients stored in the coefficient RAM 5 are described below.

The predictive-computation circuit 6 performs product-sum computationexpressed by equation (1) by using the tap coefficients w₁, w₂, . . .for the class of the main pixel supplied from the coefficient RAM 5 andthe pixels values x₁, x₂, . . . of the SD pixels forming the predictivetap supplied from the predictive-tap generating circuit 3 so as todetermine the predictive value of the main pixel y. Thepredictive-computation circuit 6 then supplies the predictive value tothe frame memory 10 and stores it therein as the pixel value of the HDpixel with reduced blurring.

In this embodiment, HD pixels forming each block divided from an HDimage contain at least one main pixel and pixel other than the mainpixel, i.e., a sub pixel. The main pixel is the HD pixel determined byusing a tap coefficient obtained by learning the relationship betweensupervisor data and learner data without giving a constraint conditionto the supervisor data. The sub-pixel is the HD pixel determined byusing a tap coefficient obtained by learning the relationship betweensupervisor data and learner data by giving a constraint condition to thesupervisor data.

In this embodiment, between the two HD pixels forming the block of theHD image shown in FIG. 3, the upper HD pixel y⁽¹⁾ is the main pixel, andthe lower HD pixel y⁽²⁾ is the sub pixel.

The sub-pixel processing circuit 12, which is formed of a coefficientRAM 7, a predictive-computation circuit 8, and a decoding circuit 9,determines the predictive value of the sub pixel y⁽²⁾ of the HD pixelsforming the specified block, supplies the determined sub pixel to theframe memory 10, and stores it at the address corresponding to theposition of the sub pixel.

That is, the coefficient RAM 7 stores tap coefficients obtained bylearning the relationship between the supervisor data, which is the HDimage data as the supervisor, and the learner data, which is the SDimage data as the learner, for each of at least one class by giving aconstraint condition to the supervisor data. Then, upon receiving theclass code of the specified block from the classification circuit 4, thecoefficient RAM 7 reads the tap coefficient stored at the addresscorresponding to the class code so as to obtain the tap coefficient ofthe class of the sub pixel of the HD pixels forming the specified block,and supplies the tap coefficient to the predictive-computation circuit6. As stated above, since the same class tap is generated for the mainpixel and the sub pixel of the same block, the class of the main pixelis the same as the class of the sub pixel. Details of the learningmethod for the tap coefficients stored in the coefficient RAM 7 aredescribed below.

The predictive-computation circuit 8 performs product-sum computationcorresponding to equation (1) by using the tap coefficients Δw1, Δw2, .. . for the classes of the sub pixels that are supplied from thecoefficient RAM 7 and the pixel values of the SD pixels forming thepredictive taps x1, x2, . . . supplied from the predictive-tapgenerating circuit 3 so as to determine the predictive value of thedifference Δy between the sub pixel and the HD pixel stored in the framememory 10, and supplies the determined difference to the decodingcircuit 9. In this embodiment, it is assumed that, for the sub pixely⁽²⁾ of the specified block, for example, the difference Δy(=y⁽²⁾−y⁽¹⁾)between the sub pixel y⁽²⁾ and the main pixel y⁽¹⁾ of the specifiedblock can be determined in the predictive-computation circuit 8.

The decoding circuit 9 decodes the difference for the sub pixel suppliedfrom the predictive-computation circuit 8 into a sub pixel. Morespecifically, in this embodiment, as the difference Δy for the sub pixely⁽²⁾, the difference between the sub pixel y⁽²⁾ and the main pixel y⁽¹⁾of the same block is used. Accordingly, the decoding circuit 9 reads thepredictive value of the main pixel y⁽¹⁾ of the specified block from theframe memory 10, and adds the main pixel y⁽¹⁾ to the difference Δysupplied from the predictive-computation circuit 8, thereby determiningthe predictive value of the sub pixel y⁽²⁾ (=Δy+y⁽¹⁾). The decodingcircuit 9 then supplies the determined sub pixel to the frame memory 10and stores it therein.

The image conversion processing for converting an SD image into an HDimage performed by the image processing apparatus shown in FIG. 2 isdescribed below with reference to the flowchart of FIG. 4.

An SD image (moving picture) to be subjected to image conversionprocessing is supplied to the frame memory 1 sequentially in units offrames, and the frames of the SD image are sequentially stored in theframe memory 1.

In step S1, the class-tap generating circuit 2 specifies a frame (whichis to be stored in the frame memory 10) of an HD image to be determined,and forms the specified frame into blocks, each block consisting of twovertically adjacent HD pixels y⁽¹⁾ and y⁽²⁾, as discussed above withreference to FIG. 3. The process then proceeds to step S2.

In step S2, the class-tap generating circuit 2 specifies one of theundetermined blocks forming the specified frame, and proceeds to stepS3.

In step S3, the class-tap generating circuit 2 and the predictive-tapgenerating circuit 3 extract a plurality of SD pixels spatially ortemporally adjacent to the position of the specified block from the SDimage stored in the frame memory 1 so as to generate a class tap and apredictive tap, respectively. Also in step S3, the class-tap generatingcircuit 2 supplies the class tap of the specified block to theclassification circuit 4, and the predictive-tap generating circuit 3supplies the predictive tap of the specified block to the main-pixelprocessing circuit 11 and the sub-pixel processing circuit 12. Theprocess then proceeds to step S4.

In step S4, the classification circuit 4 classifies each of the mainpixel y⁽¹⁾ and the sub pixel y⁽²⁾ forming the specified block based onthe class tap supplied from the class-tap generating circuit 2, andsupplies the class code representing the resulting class for thespecified block to the main-pixel processing circuit 11 and thesub-pixel processing circuit 12. The process then proceeds to step S5.

In step S5, the coefficient RAM 5 of the main-pixel processing circuit11 reads the tap coefficient stored at the address corresponding to theclass code supplied from the classification circuit 4 so as to obtainthe tap coefficient of the class of the main pixel y⁽¹⁾ forming thespecified block, and supplies the tap coefficient to thepredictive-computation circuit 6. Also in step S5, the coefficient RAM 7of the sub-pixel processing circuit 12 reads the tap coefficient storedat the address corresponding to the class code supplied from theclassification circuit 4 so as to obtain the tap coefficient of theclass of the sub pixel y⁽²⁾ forming the specified block, and suppliesthe tap coefficient to the predictive-computation circuit 8.

Thereafter, the process proceeds to step S6 in which the main-pixelprocessing circuit 11 specifies the main pixel of the specified block,and calculates the predictive value of the specified pixel y⁽¹⁾. Morespecifically, in step S6, in the main-pixel processing circuit 11, thepredictive-computation circuit 6 performs computation expressed byequation (1) by using the predictive tap supplied from thepredictive-tap generating circuit 3 and the tap coefficient suppliedfrom the coefficient RAM 5, thereby determining the predictive value ofthe main pixel y⁽¹⁾, which is the specified pixel. Also in step S6, thepredictive-computation circuit 6 supplies the determined main pixel y⁽¹⁾to the frame memory 10, and stores it at the address corresponding tothe position of the main pixel y⁽¹⁾. The process then proceeds to stepS7.

In step S7, the sub-pixel processing circuit 12 specifies the sub pixelof the specified block, and calculates the predictive value of thespecified pixel y⁽²⁾. More specifically, in step S7, in the sub-pixelprocessing circuit 12, the predictive-computation circuit 8 performsproduct-sum computation (details are given below) corresponding toequation (1) by using the predictive tap supplied from thepredictive-tap generating circuit 3 and the tap coefficient suppliedfrom the coefficient RAM 7 so as to determine the predictive value ofthe difference Δy between the sub pixel y⁽²⁾, which is the specifiedpixel, and the HD pixel y⁽¹⁾, which is the main pixel, and supplies thedifference to the decoding circuit 9. Also in step S7, the decodingcircuit 9 adds the main pixel y⁽¹⁾ determined in step S6 and stored inthe frame memory 10 to the difference Δy supplied from thepredictive-computation circuit 8, thereby determining the predictivevalue of the sub pixel y⁽²⁾, which is the specified pixel. The decodingcircuit 9 then supplies the sub pixel y⁽²⁾ to the frame memory 10, andstores it at the address corresponding to the position of the sub pixely⁽²⁾. The process then proceeds to step S8.

In step S8, the class-tap generating circuit 2 determines whether thereis any block that has not been determined as the specified block formingthe specified frame. If it is determined that there is such a block, theprocess returns to step S2, and processing similar to theabove-described processing is repeated.

If it is determined that there is no block that has not been determinedas the specified block, i.e., that all the HD pixels forming thespecified frame are stored in the frame memory 10, the process proceedsto step S9. In step S9, the frame memory 10 reads and outputs the HDimage of the specified frame, and the process then proceeds to step S10.

In step S10, the class-tap generating circuit 2 determines whether thereis any frame of the HD image to be subsequently determined. If it isdetermined that there is such a frame, the process returns to step S1 inwhich the frame is specified, and processing similar to theabove-described processing is repeated.

If it is determined in step S10 that there is no frame of the HD imageto be subsequently determined, the process is completed.

FIG. 5 illustrates an example of the configuration of an embodiment of alearning device that conducts learning for determining a tap coefficientfor each class to be stored in the coefficient RAM 5 shown in FIG. 2.

In a learning database 21, for example, HD image data, as image data forlearning tap coefficients is stored.

A learning-pair generating circuit 22 generates learning-pair data as aset of supervisor data and learner data used for learning tapcoefficients from the learning image data stored in the learningdatabase 21, and supplies the learning-pair data to a learning-pairdatabase 63.

That is, the learning-pair generating circuit 22 reads the learningimage data stored in the learning database 21, and sets the HD imagedata, which is the learning image data, directly as supervisor data. TheHD image obtained in the image processing apparatus shown in FIG. 2 isequivalent to the image quality of the HD image data used as thesupervisor data in the learning device shown in FIG. 5.

The learning-pair generating circuit 22 also decreases the image qualityof the HD image data as the supervisor data, and more specifically, thelearning-pair generating circuit 22 reduces, for example, the number ofpixels of the HD image data as the supervisor data, and then filters theHD image data having a reduced number of pixels with a LPF (Low PassFilter), thereby generating learner data, which is the SD image data,determined by blurring the HD image data as the supervisor data. The SDimage data as the learner data must coincide with the image quality ofthe SD image data to be processed in the image processing apparatusshown in FIG. 2.

The learning-pair generating circuit 22 generates the supervisor dataand the corresponding learner data generated from the supervisor data asdescribed above, and supplies a set of the supervisor data and thelearner data to the learning-pair database 23 as the learning-pair data.

The learning-pair database 23 temporarily stores the learning-pair datasupplied from the learning-pair generating circuit 22 therein.

A class-tap generating circuit 24 forms the HD image, which serves asthe supervisor data of the learning-pair data stored in thelearning-pair database 23, into the same blocks as those in theclass-tap generating circuit 2 shown in FIG. 2, and sequentiallyspecifies each block. The class-tap generating circuit 24 also generatesa class tap for each of the main pixel and the sub pixel forming thespecified block from the SD image as the learner data of thelearning-pair data stored in the learning-pair database 23, and suppliesthe class tap to a classification circuit 26. The class-tap generatingcircuit 24 generates a class tap having the same tap structure as thatgenerated by the classification circuit 2 shown in FIG. 2.

A predictive-tap generating circuit 25 generates a predictive tap foreach of the main pixel and the sub pixel forming the specified blockfrom the SD image as the learner data of the learning-pair data storedin the learning-pair database 23, and supplies the predictive tap to asumming circuit 27. The predictive-tap generating circuit 25 generates aclass tap having the same tap structure as that generated by thepredictive-tap generating circuit 3 shown in FIG. 2.

A classification circuit 26, which is configured similarly to theclassification circuit 4 shown in FIG. 2, classifies each of the mainpixel and the sub pixel forming the specified block based on the classtap supplied from the class-tap generating circuit 24, and supplies theclass code representing the class of the specified block to the summingcircuit 27.

By using the supervisor data, which is the main pixel of the specifiedblock, of the learning-pair data stored in the learning-pair database 23and the predictive tap supplied from the predictive-tap generatingcircuit 25, the summing circuit 27 and a tap-coefficient computingcircuit 28 conduct learning for the relationship between the supervisordata and the learner data as the learning-pair data stored in thelearning-pair database 23 for each class supplied from theclassification circuit 26, thereby determining a tap coefficient foreach class.

That is, the summing circuit 27 performs summation in equations (8) forthe predictive taps supplied from the predictive-tap generating circuit25 and the HD pixels serving as the supervisor data, which is the mainpixel of the specified block of the learning-pair data stored in thelearning-pair database 23 for each class code output from theclassification circuit 26.

More specifically, the summing circuit 27 performs calculationscorresponding to multiplication (x_(n,k)x_(n′,k)) of the SD pixels andsummation (Σ) in the matrix at the left side of equations (8) by usingthe SD pixels x_(n,k) serving as the learner data forming the predictivetap for each class corresponding to the class code supplied from theclassification circuit 26.

The summing circuit 27 also performs calculations corresponding tomultiplication (x_(n,k)y_(k)) of the SD pixels x_(n,k) and the HD pixely_(k) and summation (Σ) in the vector at the right side of equations (8)by using the SD pixels x_(n,k) serving as the learner data forming thepredictive tap and the HD pixel y_(k) as the supervisor data, which isthe main pixel of the specified block, for each class corresponding tothe class code supplied from the classification circuit 26.

When establishing the normal equations expressed in equations (8) foreach class by performing the above-described summation by using all theblocks of the HD image as the supervisor data in the learning-pair datastored in the learning-pair database 23, the summing circuit 27 suppliesthe normal equations to the tap-coefficient computing circuit 28.

Upon receiving the normal equations expressed by equations (8) for eachclass from the summing circuit 27, the tap-coefficient computing circuit28 solves the normal equations to determine the tap coefficient for eachclass, and outputs the tap coefficient.

A tap-coefficient memory 29 stores the tap coefficient for each classsupplied from the tap-coefficient computing circuit 28.

FIG. 6 illustrates an example of the configuration of the learning-pairgenerating circuit 22 shown in FIG. 5.

HD image data as the learning image data stored in the learning database21 (FIG. 5) is supplied to the learning-pair generating circuit 22, andthe learning-pair generating circuit 22 outputs the HD image data as thesupervisor data.

A decimation circuit 31 reduces the number of pixels of the HD imagedata as the learning image data, and supplies the resulting SD imagedata to an LPF 32. As discussed with reference to FIG. 3, in thisembodiment, SD image data is converted into HD image data having thenumber of vertical and horizontal pixels twice as that of the SD imagedata in the image processing apparatus 2. Accordingly, conversely, inthe decimation circuit 31, the number of vertical and horizontal pixelsof the HD image data as the learning image data is reduced to one half.

The LPF 32, having predetermined frequency characteristics, filters theSD image data supplied from the decimation circuit 31 so as to obtainthe blurred SD image data, and outputs it as the learner data.

The learning-pair generating circuit 22 outputs a set of the supervisordata and the learner data obtained as described above to thelearning-pair database 23 (FIG. 5) as the learning-pair data.

Learning processing for determining a tap coefficient for each classperformed in the learning device shown in FIG. 5 is described below withreference to the flowchart of FIG. 7.

In step S11, the learning-pair generating circuit 22 reads learningimage data from the learning database 21 so as to generate supervisordata and learner data. Also in step S11, the learning-pair generatingcircuit 22 generates learning-pair data by forming a set of thesupervisor data and the learner data, supplies the learning-pair data tothe learning-pair database 23, and stores it therein.

The process then proceeds to step S12. In step S12, the class-tapgenerating circuit 24 forms the HD image data as the supervisor data ofthe learning-pair data stored in the learning-pair database 23 intoblocks, each block consisting of two vertically adjacent HD pixels, in amanner similar to the class-tap generating circuit 2 shown in FIG. 2.The process then proceeds to step S13.

In step S13, the class-tap generating circuit 24 specifies one of theundetermined blocks of the HD image serving as the supervisor data ofthe learning-pair data stored in the learning-pair database 23, and theprocess proceeds to step S14. In step S14, the class-tap generatingcircuit 24 and the predictive-tap generating circuit 25 generate a classtap and a predictive tap for the main pixel of the specified block fromthe SD pixels as the learner data stored in the learning-pair database23, and supplies the class tap and the predictive tap to theclassification circuit 26 and the summing circuit 27, respectively. Theprocess then proceeds to step S15.

In step S15, as in the classification circuit 4 shown in FIG. 2, theclassification circuit 26 classifies the main pixel of the specifiedblock by using the class tap supplied from the class-tap generatingcircuit 24, and supplies the class code representing the resulting classfor the specified block to the summing circuit 27. The process thenproceeds to step S16.

In step S16, the summing circuit 27 reads the HD pixel, which is themain pixel of the specified block, from the learning-pair database 23 asthe specified pixel. Also in step S16, the summing circuit 27 performssummation in equations (8) for the predictive taps supplied from thepredictive-tap generating circuit 25 and the specified pixels read fromthe learning-pair database 23 for each class of the specified blockrepresented by the class code supplied from the classification circuit26. The process then proceeds to step S17.

In step S17, the class-tap generating circuit 24 determines whetherthere is any unspecified block in the HD image data serving as thesupervisor data stored in the learning-pair database 23. If it isdetermined in step S17 that there is an unspecified block in thesupervisor data stored in the learning-pair database 23, the processreturns to step S13, and processing similar to the above-describedprocessing is repeated.

If it is determined in step S17 that there is no unspecified block inthe supervisor data stored in the learning-pair database 23, the summingcircuit 27 supplies the normal equations in equations (8) obtained bysumming in step S16 for each class to the tap-coefficient computingcircuit 28. The process then proceeds to step S18.

In step S18, the tap-coefficient computing circuit 28 solves the normalequations in equations (8) for each class supplied from the summingcircuit 27 so as to determine a tap coefficient for each class, suppliesthe tap coefficient to the tap-coefficient memory 29, and stores ittherein. The process is then completed.

The tap coefficient for each class stored in the tap coefficient memory29 as described above is the same tap coefficient stored in thecoefficient RAM 5 of the image processing apparatus shown in FIG. 2. Inthe main-pixel processing circuit 11 provided with the coefficient RAM5, HD pixels can be determined as the main pixels in a manner similar tothe classification adaptive processing proposed by the presentapplicant.

In the above-described learning processing for tap coefficients, theremay be a class for which a required number of normal equations fordetermining a tap coefficient cannot be obtained depending on learningimage data prepared. For such a class, for example, a default tapcoefficient may be output from the tap-coefficient computing circuit 28.Alternatively, if there is any class for which a required number ofnormal equations for determining a tap coefficient cannot be obtained,new learning image data may be prepared to re-conduct learning for tapcoefficients. The same applies to learning for tap coefficient in alearning device, which is described below.

FIG. 8 illustrates an example of the configuration of an embodiment of alearning device for conducting learning for determining a tapcoefficient for each class stored in the coefficient RAM 7 shown in FIG.2.

The learning device shown in FIG. 8 includes a learning database 41, alearning-pair generating circuit 42, a learning-pair database 43, aclass-tap generating circuit 44, a predictive-tap generating circuit 45,a classification circuit 46, a summing circuit 47, a tap-coefficientcomputing circuit 48, and a tap-coefficient memory 49, which areconfigured similarly to the learning database 21, the learning-pairgenerating circuit 22, the learning-pair database 23, the class-tapgenerating circuit 24, the predictive-tap generating circuit 25, theclassification circuit 26, the summing circuit 27, the tap-coefficientcomputing circuit 28, and the tap-coefficient memory 29, respectively,shown in FIG. 5.

Basically, therefore, in the learning device shown in FIG. 8, a tapcoefficient for each of at least one class is determined in a mannersimilar to the learning device shown in FIG. 5. In the learning deviceshown in FIG. 8, however, the relationship between supervisor data andlearner data is learned for each of at least one class by giving apredetermined constraint condition to the supervisor data, therebydetermining a tap coefficient for each of at least one class.

That is, in the learning device shown in FIG. 8, the relationshipbetween supervisor data and learner data is learned by giving aconstraint condition that imposes constraint on the relationship betweenthe main pixel and the sub pixel of a block forming the supervisor data.More specifically, by giving a constraint condition that imposesconstraint on the difference between the main pixel and the sub pixel,the relationship between the supervisor data and the learner data islearned.

By learning the relationship between the supervisor data and the learnerdata while imposing constraint on the difference Δy between the mainpixel y⁽¹⁾ and the sub pixel y⁽²⁾, that is, by learning the relationshipbetween the supervisor data and the learner data ideally under theconstraint condition that the difference Δy′=y⁽²⁾′−y⁽¹⁾′ between thepredictive value y⁽¹⁾′ of the main pixel and the predictive value y⁽²⁾′of the sub pixel determined in the product-sum computation in equation(1) coincides with the true value Δy=y⁽²⁾−y⁽¹⁾ of the differenceobtained by subtracting the true value y⁽¹⁾ of the main pixel from thetrue value y⁽²⁾ of the sub pixel, the tap coefficient for determiningthe predictive value y⁽²⁾′ of the sub pixel while maintaining therelationship between the true value y⁽¹⁾ of the main pixel and the truevalue y⁽²⁾ of the sub pixel can be obtained.

When an SD image is converted into an HD image by using such a tapcoefficient, the situation in which a change in the pixel value becomesopposite to a change in the true value, as discussed with reference toFIG. 1B, can be prevented due to the effect of the constraint condition,i.e., by imposing constraint on the difference between the main pixeland the sub pixel.

It is now assumed that the condition that the difference Δy′=y⁽²⁾′−y⁽¹⁾′between the predictive value y⁽¹⁾′ of the main pixel and the predictivevalue y⁽²⁾′ of the sub pixel (hereinafter sometimes referred to as the“predictive value of the difference”) coincides with the true valueΔy=y⁽²⁾−y⁽¹⁾ of the difference between the true value y⁽¹⁾ of the mainpixel and the true value y⁽²⁾ of the sub pixel is used as the constraintcondition for imposing constraint on the difference Δy obtained bysubtracting the main pixel y⁽¹⁾ from the sub pixel y⁽²⁾. In actuality,however, it is difficult to satisfy such a constraint condition for themain pixels and the sub pixels of all the blocks.

In the learning device shown in FIG. 8, therefore, the relationshipbetween the supervisor data and the learner data is learned under theconstraint condition that the predictive error of the predictive valueof the difference Δ′y with respect to the true value Δy is statisticallycontained to a minimal value.

That is, when the tap coefficients for determining the main pixel y⁽¹⁾and the sub pixel y⁽²⁾ are indicated by w_(n) ⁽¹⁾ and w_(n) ⁽²⁾,respectively, the main pixel y⁽¹⁾ and the sub pixel y⁽²⁾ can bedetermined by using the tap coefficients w_(n) ⁽¹⁾ and w_(n) ⁽²⁾ and thepredictive tap x_(n) according to equation (1) by using equations (9)and (10), respectively. $\begin{matrix}{y^{(1)} = {\sum\limits_{n = 1}^{N}{w_{n}^{(1)}x_{n}}}} & (9) \\{y^{(2)} = {\sum\limits_{n = 1}^{N}{w_{n}^{(2)}x_{n}}}} & (10)\end{matrix}$

The difference Δw_(n) between the tap coefficients w_(n) ⁽¹⁾ and w_(n)⁽²⁾ is defined by the following equation.Δw _(n) =w _(n) ⁽²⁾ −w _(n) ⁽¹⁾  (11)

In this case, the difference Δy is determined by the followingproduct-sum computation from equations (9) through (11). $\begin{matrix}{{\Delta\quad y} = {\sum\limits_{n = 1}^{N}{\Delta\quad w_{n}x_{n}}}} & (12)\end{matrix}$

The constraint condition that the predictive error of the predictivevalue of the difference Δ′y with respect to the true value Δy isstatistically minimized can be satisfied, for example, by the conditionthat the sum of the predictive errors of the predictive values of thedifferences Δ′y is minimized, and the tap coefficient Δw_(n) thatminimizes the sum of the predictive errors is a tap coefficientsatisfying the constraint condition.

The tap coefficient Δw_(n) that minimizes the sum of the predictiveerrors of th e predictive value s of the differences Δ′y can bedetermined by, for example, the method of least squares.

Equation (12) is equivalent to the equation in which the HD pixel y inequation (1) is substituted by the difference Δy and the tap coefficientw_(n) is substituted by the tap coefficient Δw_(n). Accordingly, the tapcoefficient Δw_(n) that minimizes the sum of the predictive errors ofthe differences y determined in equation (12) can be determined bysolving normal equations in equations (13) that are obtained bysubstituting the HD pixel y in equation (8) by the difference Δy and bysubstituting the tap coefficient w_(n) by the tap coefficient Δw_(n).$\begin{matrix}{{\begin{bmatrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{N,k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{N,k}}} \right) \\\vdots & \vdots & ⋰ & \vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{2,k}}} \right) & \cdots & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{N,k}}} \right)\end{bmatrix}\left\lbrack \quad\begin{matrix}{\Delta\quad w_{1}} \\{\Delta\quad w_{2}} \\\vdots \\{\Delta\quad w_{N}}\end{matrix} \right\rbrack}\quad{\begin{matrix} = \\ = \\ = \end{matrix}\left\lbrack \quad\begin{matrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}\Delta\quad y_{k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}\Delta\quad y_{k}}} \right) \\\vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}\Delta\quad y_{k}}} \right)\end{matrix}\quad \right\rbrack}} & (13)\end{matrix}$

In the learning device shown in FIG. 8, the relationship between thesupervisor data y_(k) and the learner data x_(k) is learned for each ofat least one class by giving the above-described constraint condition tothe supervisor data y_(k), thereby determining the tap coefficientΔw_(n) for each of at least one class.

Accordingly, the learning performed in the learning device shown in FIG.8 by using the difference Δy of the main pixel y⁽¹⁾ and the sub pixely⁽²⁾ of the block as the constrained supervisor data with theconstrained condition (hereinafter sometimes referred to as “constrainedsupervisor data”) is equivalent to the learning of the relationshipbetween the constrained supervisor data Δy and the learner data x_(k).

Accordingly, in the learning-pair generating circuit 42, a set ofconstrained supervisor data and learner data is generated aslearning-pair data rather than a set of supervisor data itself andlearner data. In the summing circuit 47, summing in equations (13) forthe constrained supervisor data and the learner data is performed ratherthan summing in equations (8) for the supervisor data and the learnerdata.

FIG. 9 illustrates an example of the configuration of the learning-pairgenerating circuit 42 shown in FIG. 8.

HD image data serving as learning image data stored in the learningdatabase 41 (FIG. 8) is supplied to the learning-pair generating circuit42, and the learning-pair generating circuit 42 supplies the HD imagedata to a decimation circuit 51, a memory 53, a computation circuit 54,and a selector 55 as the supervisor data.

As in the decimation circuit 31 shown in FIG. 6, the decimation circuit51 reduces the number of pixels of the HD image data as the supervisordata, and supplies the resulting SD image data to an LPF 52. The LPF 52having predetermined frequency characteristics, similar to the LPF 32shown in FIG. 6, filters the SD image data supplied from the decimationcircuit 51 so as to obtain the blurred SD image data, and outputs the SDpixels forming the SD image data serving as the learner data.

The memory 53 temporarily stores the HD image data as the supervisordata supplied from the learning database 41, and supplies the HD pixelto serve as the main pixel y⁽¹⁾ of a block when being divided from theHD image data to the computation circuit 54.

The computation circuit 54 receives the HD pixel to serve as the subpixel y⁽²⁾ of the block when being divided from the HD image data as thesupervisor data supplied from the learning database 41. The computationcircuit 54 also subtracts the main pixel y⁽¹⁾ of the block supplied fromthe memory 53 from the sub pixel y⁽²⁾ of the same block so as todetermine the difference Δy=y⁽²⁾−y⁽¹⁾, and outputs the difference Δy tothe selector 55.

As for the HD image data to serve as the main pixel of the block whenbeing divided from the HD image data as the supervisor data suppliedfrom the learning database 41, the selector 55 selects the HD pixel asit is. As for the HD pixel to serve as the sub pixel, the selectorselects the difference Δy output from the computation circuit 54. Themain pixel and the sub pixel are then output as the constrainedsupervisor data.

Accordingly, as the constrained supervisor data output from the selector55, the main pixel has the pixel value of the HD pixel itself as thesupervisor data, and the sub pixel has the pixel value obtained bysubtracting the pixel value of the main pixel from the pixel value ofthe sub pixel.

The learning-pair generating circuit 42 outputs a set of the constrainedsupervisor data and the learner data obtained as described above to thelearning-pair database 43 (FIG. 8) as the learning-pair data.

Learning processing for determining a tap coefficient for each classperformed in the learning device in FIG. 8 is described below withreference to the flowchart of FIG. 10.

In step S21, the learning-pair generating circuit 42 reads learningimage data from the learning database 41, and generates constrainedsupervisor data and learner data. Also in step S21, the learning-pairgenerating circuit 42 generates learning-pair data formed of a set ofthe constrained supervisor data and the learner data, supplies thelearning-pair data to the learning-pair database 43, and stores ittherein.

The process then proceeds to step S22. In step S22, the class-tapgenerating circuit 44 divides the constrained supervisor data of thelearning-pair data stored in the learning-pair database 43 into blocks,each block consisting of two vertically adjacent HD pixels, in a mannersimilar to the class-tap generating circuit 2 shown in FIG. 2. Theprocess then proceeds to step S23.

In step S23, the class-tap generating circuit 44 specifies one of theundetermined blocks of the constrained supervisor data of thelearning-pair data stored in the learning-pair database 43, and theprocess proceeds to step S24. In step S24, the class-tap generatingcircuit 44 and the predictive-tap generating circuit 45 generate a classtap and a predictive tap, respectively, for the sub pixel of thespecified block from the SD pixels as the learner data stored in thelearning-pair database 43, and supplies the class tap and the predictivetap to the classification circuit 46 and the summing circuit 47,respectively. The process then proceeds to step S25.

In step S25, as in the classification circuit 4 shown in FIG. 2, theclassification circuit 46 classifies the sub pixel of the specifiedblock by using the class tap supplied from the class-tap generatingcircuit 44, and supplies the class code representing the resulting classof the specified block to the summing circuit 47. The process thenproceeds to step S26.

In step S26, the summing circuit 47 reads the difference Δy of the pixelvalue of the HD pixel, which serves as the sub pixel of the specifiedblock, from the learning-pair database 43 as the specified pixel. Alsoin step S26, the summing circuit 47 performs summation in equations (13)for the predictive taps supplied from the predictive-tap generatingcircuit 45 and the specified pixels read from the learning-pair database43 for each class of the specified block represented by the class codesupplied from the classification circuit 46. The process then proceedsto step S27.

In step S27, the class-tap generating circuit 44 determines whetherthere is any unspecified block in the blocks of the constrainedsupervisor data stored in the learning-pair database 43. If it isdetermined in step S27 that there is an unspecified block in the blocksof the constrained supervisor data stored in the learning-pair database43, the process returns to step S23, and processing similar to theabove-described processing is repeated.

If it is determined in step S27 that there is no unspecified block inthe blocks of the constrained supervisor data stored in thelearning-pair database 43, the summing circuit 47 supplies the normalequations (13) obtained for each class by performing summation in stepS26 to the tap-coefficient computing circuit 48. The process thenproceeds to step S28.

In step S28, the tap-coefficient computing circuit 48 solves the normalequations in (13) for each class supplied from the summing circuit 47 soas to determine the tap coefficient Δw_(n) for each class, and suppliesthe tap coefficient to the tap-coefficient memory 49 and stores ittherein. The process is then completed.

The tap coefficient Δw_(n) for each class stored in the tap-coefficientmemory 49 as described above is the same tap coefficient stored in thecoefficient RAM 7 of the image processing apparatus shown in FIG. 2.Then, in the sub-pixel processing circuit 12 provided with thecoefficient RAM 7, by performing product-sum computation of equation(12) corresponding to equation (1) by using the tap coefficient Δw_(n)stored in the coefficient RAM 7 and the predictive tap x_(n), thepredictive-computation circuit 8 determines the predictive value of thedifference Δy of the sub pixel y⁽²⁾ to the main pixel y⁽¹⁾ thatminimizes the sum of the predictive errors. In the decoding circuit 9,the difference Δy is added to the predictive value of the main pixely⁽¹⁾ stored in the frame memory 10 so as to determine the predictivevalue of the sub pixel y⁽²⁾.

Accordingly, since the sub pixel is determined by using the tapcoefficient that can maintain the relationship between the sub pixel andthe main pixel, a high-quality HD image can be obtained in the imageprocessing apparatus shown in FIG. 2 by preventing the situation inwhich a change in the pixel value of the HD image to be determinedbecomes opposite to a change in the true value.

FIG. 11 illustrates another example of the configuration of thelearning-pair generating circuit 42 shown in FIG. 8. In FIG. 11, theelements corresponding to those of FIG. 9 are designated with likereference numerals, and an explanation thereof is thus omitted. That is,the learning-pair generating circuit 42 shown in FIG. 11 is configuredsimilarly to that of FIG. 9, except that a new element, i.e., aclassification-adaptive processing circuit 60, is disposed.

SD image data as learner data output from the LPF 52 is supplied to theclassification-adaptive processing circuit 60. Theclassification-adaptive processing circuit 60 performs classificationadaptive processing previously proposed by the present applicant on theSD image data as the learner data so as to determine the predictivevalue of the HD image data as the supervisor data, and supplies the HDimage data to the memory 53.

More specifically, the classification-adaptive processing circuit 60includes a frame memory 61, a class-tap generating circuit 62, apredictive-tap generating circuit 63, a classification circuit 64, acoefficient RAM 65, and a predictive-computation circuit 66, which aresimilarly configured to the frame memory 1, the class-tap generatingcircuit 2, the predictive-tap generating circuit 3, the classificationcircuit 4, the coefficient RAM 5, and the predictive-computation circuit6, respectively, shown in FIG. 2. As in the image processing apparatusshown in FIG. 2, the classification-adaptive processing circuit 60performs classification adaptive processing for converting SD image dataas the learner data supplied from the LPF 52 into HD image data, andsupplies the resulting HD image data to the memory 53.

In the classification-adaptive processing circuit 60, however,classification adaptive processing is performed, assuming that all theblocks of the HD image data consist of main pixels. That is, in theclassification-adaptive processing circuit 60, the tap coefficientsdetermined by the learning device shown in FIG. 5 are stored in thecoefficient RAM 65, and all the HD pixels forming the blocks of the HDimage data are determined by performing product-sum computation ofequation (1) using the tap coefficients stored in the coefficient RAM65.

Accordingly, in the embodiment of FIG. 11, the predictive values of thesupervisor data determined by the classification-adaptive processingcircuit 60 are stored in the memory 53 rather than the supervisor dataitself as the learning image data stored in the learning database 41. Asa result, in the computation circuit 54 shown in FIG. 11, the differenceΔy is determined by subtracting the predictive value of the main pixely⁽¹⁾ of the block divided from the HD image data as the supervisor datasupplied from the learning database 41 and stored in the memory 53 fromthe HD pixel serving as the sub pixel y⁽²⁾ of the same block. That is,the difference Δy is determined as the constrained supervisor data bysubtracting, not the main pixel y⁽¹⁾ itself of the block, but thepredictive value of the main pixel y⁽¹⁾ determined by theclassification-adaptive processing circuit 60, from the HD pixel servingas the sub pixel y⁽²⁾ of the same block.

Thus, tap coefficients having higher prediction precision for sub pixels(having smaller predictive errors for sub pixels) can be determinedcompared to when the learning-pair generating circuit 42 is configured,as shown in FIG. 9.

More specifically, in the image processing apparatus shown in FIG. 2,the predictive value of the difference Δy between the sub pixel y⁽²⁾ andthe main pixel y⁽¹⁾ is determined by performing product-sum computationof equation (12) using the predictive tap x_(k) and the tap coefficientΔw_(n) in the predictive-computation circuit 8 of the sub-pixelprocessing circuit 12. Then, in the decoding circuit 9, the predictivevalue of the main pixel y⁽¹⁾ determined in the main-pixel processingcircuit 11 is added to the predictive value of the difference ydetermined in the predictive-computation circuit 8, thereby determiningthe predictive value of the sub pixel y⁽²⁾.

Not the true value of the main pixel, but the predictive value of themain pixel is used for determining the sub pixel in the image processingapparatus shown in FIG. 2. Accordingly, also in the learning-pairgenerating circuit 42 of the learning device shown in FIG. 8, if thepredictive value of the main pixel is used for generating theconstrained supervisor data rather than the true value of the mainpixel, the prediction precision of the sub pixel in the image processingapparatus shown in FIG. 2 can be improved.

In the embodiment shown in FIG. 11, the predictive values of the all theHD pixels of the HD image data as the supervisor data are determined inthe classification-adaptive processing circuit 60. In the memory 53 andthe computation circuit 54, which are the subsequent stage of theclassification-adaptive processing circuit 60, among the HD pixelsoutput from the classification-adaptive processing circuit 60, only theHD pixels to serve as the main pixels in the image processing apparatusin FIG. 2 are used. Accordingly, in the classification-adaptiveprocessing circuit 60, only HD pixels to serve as main pixels in theimage processing apparatus in FIG. 2 may be determined, and HD pixels toserve as sub pixels may be ignored (not processed).

FIG. 12 illustrates an example of the configuration of anotherembodiment of the image processing apparatus to which the presentinvention is applied. In FIG. 12, the elements corresponding to thoseshown in FIG. 2 are designated with like reference numerals, and anexplanation thereof is thus omitted. That is, the image processingapparatus shown in FIG. 12 is configured similarly to that shown in FIG.2, except that the sub-pixel processing circuit 12 is formed of acoefficient RAM 77 and a predictive-computation circuit 78 instead ofthe coefficient RAM 7, the predictive-computation circuit 8, and thedecoding circuit 9.

In the coefficient RAM 77, a tap coefficient for each class obtained byadding the tap coefficient w_(n) for each class stored in thecoefficient RAM 5 shown in FIG. 2 to the tap coefficient Δw_(n) storedin the coefficient RAM 7 is stored. The coefficient RAM 77 reads the tapcoefficient of the class represented by the class code of the specifiedblock supplied from the classification circuit 4 so as to obtain theread tap coefficient, and supplies it to the predictive-computationcircuit 78.

The predictive-computation circuit 78 performs product-sum computationcorresponding to equation (1) by using the tap coefficient of the classof the specified block supplied from the coefficient RAM 77 and thepredictive tap of the specified block supplied from the predictive-tapgenerating circuit 3, thereby determining the predictive value of thesub pixel of the specified block. The predictive-computation circuit 78then supplies the determined sub pixel to the corresponding address ofthe frame memory 10, and stores it therein.

That is, according to the learning operation performed by usingconstrained supervisor data, the tap coefficient Δw_(n), which is usedfor product-sum computation in equation (12), for determining thedifference Δy by subtracting the main pixel y⁽¹⁾ from the sub pixely⁽²⁾, is obtained. The tap coefficient Δw_(n) is defined in equation(11).

The tap coefficient w_(n) ⁽²⁾ can be found according to the followingequation from equation (11).w _(n) ⁽²⁾ =w _(n) ⁽¹⁾ +Δw _(n)  (14)

According to equation (14), the tap coefficient w_(n) ⁽²⁾ is obtained bythe tap coefficient Δw_(n) that is determined by performing learningusing the constrained supervisor data. Accordingly, in other words, thetap coefficient w_(n) ⁽²⁾ is determined by performing learning using theconstrained supervisor data. Then, according to the tap coefficientw_(n) ⁽²⁾, the predictive value of the sub pixel y⁽²⁾ can be determinedby product-sum computation of equation (10) corresponding to equation(1).

In the embodiment shown in FIG. 12, the tap coefficient w_(n) ⁽²⁾ foreach class expressed by equation (14) is stored in the coefficient RAM77. Then, in the predictive-computation circuit 78, product-sumcomputation of equation (10) is performed by using the tap coefficientw_(n) ⁽²⁾ and the predictive tap x_(n) output from the predictive-tapgenerating circuit 3, thereby determining the sub pixel y⁽²⁾.

Accordingly, in the image processing apparatus shown in FIG. 12, in stepS5 of FIG. 4, instead of obtaining the tap coefficient Δw_(n) from thecoefficient RAM 7, the tap coefficient w_(n) ⁽²⁾ is obtained from thecoefficient RAM 77. In step S8, in the image processing apparatus inFIG. 2, the difference Δy is determined by product-sum computation ofequation (12) using the tap coefficient Δw_(n) stored in the coefficientRAM 7 and the predictive tap x_(n), and the difference Δy and the mainpixel y⁽¹⁾ are added, thereby determining the sub pixel y⁽²⁾. Instead ofthis operation, in the image processing apparatus in FIG. 12, in stepS8, the sub pixel y⁽²⁾ is determined by product-sum computation ofequation (10) using the tap coefficient w_(n) ⁽²⁾ stored in thecoefficient RAM 77 and the predictive tap x_(n). The other steps aresimilar to those of the flowchart of FIG. 4, and thus, an explanation ofthe processing of the image processing apparatus in FIG. 12 is omitted.

As in the image processing apparatus shown in FIG. 2, in the imageprocessing apparatus shown in FIG. 12, the sub pixel is determined byusing the tap coefficient that can maintain the relationship between thesub pixel and the main pixel. Accordingly, it is possible to prevent thesituation in which a change in the pixel value of an HD image to bedetermined becomes opposite to a change in the true value of the pixelof the HD image, thereby obtaining a higher-quality HD image.

The tap coefficient w_(n) ⁽²⁾ stored in the coefficient RAM 77 may bedetermined by adding the tap coefficient w_(n) ⁽¹⁾ determined in thelearning device in FIG. 5 to the tap coefficient Δw_(n) determined inthe learning device in FIG. 8. Alternatively, the learning devices shownin FIGS. 5 and 8 may be integrally formed, and a circuit for adding thetap coefficient w_(n) ⁽¹⁾ output from the tap-coefficient computingcircuit 28 to the tap coefficient Δw_(n) output from the tap-coefficientcomputing circuit 48 may be provided for the integrated learning device,thereby determining the tap coefficient w_(n) ⁽²⁾.

In the embodiment shown in FIG. 2, in the sub-pixel processing circuit12, the difference between the main pixel and the sub pixel isdetermined by using the pixel values x⁽¹⁾ through x⁽⁹⁾ themselves of theSD pixels shown in FIG. 3 as the predictive taps. Alternatively, thedifference between the main pixel and the sub pixel may be determined byusing the differences of the pixel values x⁽¹⁾ through x⁽⁹⁾ as thepredictive taps. That is, the difference between the main pixel and thesub pixel may be determined by using the differences of, for example,two vertically adjacent SD pixels of the pixel values x⁽¹⁾ through x⁽⁹⁾of the SD pixels, i.e., x⁽¹⁾-x⁽⁴⁾, x⁽²⁾-x⁽⁵⁾, x⁽³⁾-x⁽⁶⁾, x⁽⁴⁾-x⁽⁷⁾,x⁽⁵⁾-x⁽⁸⁾, and x⁽⁶⁾-x⁽⁹⁾, as the predictive taps. In this case, however,predictive taps similar to those obtained as described above must beused in the learning device shown in FIG. 8.

In this embodiment, for the sake of simplicity, HD image data is dividedinto blocks, each block consisting of two vertically adjacent HD pixels.However, each block may consist of three or more HD pixels. That is, theblock may consist of, as shown in FIG. 13, two-row×two-column HD pixelsy⁽¹⁾, y⁽²⁾, y⁽³⁾, and y⁽⁴⁾.

In a block, the HD pixel used as the main pixel is not restricted to onepixel. For example, if a block is formed of the four HD pixels y⁽¹⁾through y⁽⁴⁾, as shown in FIG. 13, one HD pixel y⁽¹⁾ may be used as themain pixel, and the remaining three HD pixels y⁽²⁾ through y⁽⁴⁾ may beused as the sub pixels. Alternatively, for example, the two HD pixelsy⁽¹⁾ and y⁽²⁾ may be used as the main pixels, and the remaining two HDpixels y⁽³⁾ and y⁽⁴⁾ may be used as the sub pixels.

In the learning device shown in FIG. 8, the difference obtained bysubtracting the main pixel from the sub pixel is used as the constrainedsupervisor data in the learning-pair generating circuit 42.Alternatively, the difference obtained by subtracting another sub pixelfrom the sub pixel may be used as the constrained supervisor data. Forexample, it is now assumed, as shown in FIG. 13, that a block is formedof the four HD pixels y⁽¹⁾ through y⁽⁴⁾, and that one HD pixel y⁽¹⁾ isused as the main pixel, and the remaining three HD pixels y⁽²⁾ throughy⁽⁴⁾ are used as the sub pixels. In this case, for example, for the subpixel y⁽²⁾, the difference obtained by subtracting the main pixel y⁽¹⁾from the sub pixel y⁽²⁾ may be used as the constrained supervisor data;for the sub pixel y⁽³⁾, the difference obtained by subtracting the subpixel y⁽²⁾ from the sub pixel y⁽³⁾ may be used as the constrainedsupervisor data; and for the sub pixel y⁽⁴⁾, the difference obtained bysubtracting the sub pixel y⁽³⁾ from the sub pixel y⁽⁴⁾ may be used asthe constrained supervisor data. In this case, in the decoding circuit 9of the image processing apparatus shown in FIG. 2, the sub pixel y⁽²⁾ isdetermined by adding the main pixel y⁽¹⁾ stored in the frame memory 10to the difference output from the predictive-computation circuit 8; thesub pixel y⁽³⁾ is determined by adding the sub pixel y⁽²⁾ stored in theframe memory 10 to the difference output from the predictive-computationcircuit 8; and the sub pixel y⁽⁴⁾ is determined by adding the sub pixely⁽³⁾ stored in the frame memory 10 to the difference output from thepredictive-computation circuit 8.

In this embodiment, a constraint condition for imposing constraint onthe difference of the sub pixel to the single main pixel is given to thesub pixel serving as the supervisor data. However, a constraintcondition for imposing constraint on the difference of the sub pixel toeach of a plurality of pixels may be given to the sub pixel. Forexample, in the block shown in FIG. 13, if y⁽⁴⁾ is set as the sub pixel,the differences y⁽⁴⁾−y⁽¹⁾, y⁽⁴⁾−y⁽²⁾, and y⁽⁴⁾−y⁽³⁾ may be used as theconstrained supervisor data for performing summation in the summingcircuit 47 shown in FIG. 8.

In the embodiment in FIG. 2, two vertically adjacent HD pixels are usedas one block, and the upper HD pixel and the lower HD pixel are used asthe main pixel and the sub pixel, respectively. Then, the predictivevalue of the difference between the sub pixel and the main pixel isadded to the predictive value of the main pixel, thereby determining thepredictive value of the sub pixel (reproducing the sub pixel).Alternatively, each HD pixel for one frame (field) may be determined asfollows. The predictive value of each HD pixel is set as the sub pixel,and is determined by adding the predictive value of the difference ofthe sub pixel to the predictive value of the HD pixel which isadjacently disposed above the sub pixel. In this case, for an HD pixelon the topmost line, the HD pixel is set as the main pixel, and thepredictive value of the main pixel is determined. Alternatively, thetrue value of an HD pixel on the topmost line is obtained by a certainmethod.

In this embodiment, in the learning device, the relationship betweensupervisor data and learner data is learned by giving a predeterminedconstraint condition to the supervisor data, thereby obtaining a tapcoefficient that can determine the predictive value of each HD pixelreflecting the relationship of the HD pixel with the other HD pixelsboth serving as the supervisor data. In other words, the learningoperation with a constraint condition given to the supervisor datalearns the relationship between the features obtained by a plurality ofsamples (a plurality of HD pixels) of the supervisor data and thesamples of the learner data.

That is, as stated above, in the learning device shown in FIG. 8, thedifference obtained by subtracting a main pixel from the true value of asub pixel is set as the constrained supervisor data, and therelationship between the constrained supervisor data and learner data islearned. In this case, in other words, the difference obtained bysubtracting the main pixel from the sub pixel represents the features ofthe two HD pixels, i.e., the feature of the HD pixel as the main pixeland the feature of the HD pixel as the sub pixel. In the learning deviceshown in FIG. 8, therefore, the relationships between the featuresobtained from a plurality of HD pixels of the supervisor data and aplurality of SD pixels as the learner data are learned, and the tapcoefficients for associating the supervisor data with the learner dataare determined.

The features obtained from a plurality of HD pixels of the supervisordata are not restricted to the differences.

A plurality of HD pixels used for obtaining features are not restrictedto two HD pixels, and three or more HD pixels can be used.

More specifically, as the features obtained from a plurality of HDpixels of the supervisor data, differences obtained from, for example,three HD pixels, may be used. In this case, in the learning device inFIG. 8, for example, three vertically (or horizontally) adjacent HDpixels of the supervisor data may be set as a, b, and c from the top.Then, the difference Δ1=b−a obtained by subtracting the topmost HD pixela from the second HD pixel b from the top is determined. Then, thedifference Δ1 is added to the second HD pixel b from the top so as todetermine the bottommost HD pixel c, i.e., the provisional predictivevalue c′=b+Δ1. Then, the difference Δ2=c′−c obtained by subtracting thetrue value c from the provisional predictive value c′of the HD pixel cis used as the feature obtained from the three HD pixels a, b, and c.The tap coefficient that associates the difference Δ2 with thepredictive tap obtained from the learner data is learned.

In this case, in the image processing apparatus in FIG. 2, the topmostHD pixel a and the second HD pixel b from the top are used as the mainpixels, and the predictive values of the HD pixels a and b aredetermined in the main-pixel processing circuit 11. As for thebottommost HD pixel c, in the sub-pixel processing circuit 12, thepredictive value of the difference Δ1 is determined from the predictivevalues of the HD pixels a and b determined in the main-pixel processingcircuit 11, and is added to the HD pixel b determined in the main-pixelprocessing circuit 11, thereby determining the provisional predictivevalue c′ of the HD pixel c. In the sub-pixel processing circuit 12, thepredictive value of the difference Δ2 obtained from the three HD pixelsa, b, and c as the feature is determined by using the tap coefficientobtained by learning, and the difference Δ2 is subtracted from theprovisional predictive value c′ of the HD pixel c, thereby determiningthe predictive value of the HD pixel c.

The above-described series of processings may be performed by usinghardware or software. If software is used, a software program isinstalled into, for example, a general-purpose computer.

FIG. 14 illustrates an example of the configuration of an embodiment ofa computer into which a program executing the above-described series ofprocessings is installed.

A program can be prerecorded in a hard disk 105 or a ROM 103, whichserves as a recording medium integrated in the computer.

Alternatively, the program may be temporarily or permanently stored(recorded) in a removable recording medium 111, such as a flexible disk,a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk,a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductormemory. The removable recording medium 111 can be provided as so-called“package software”.

The program may be installed into the computer from the above-describedremovable recording medium 111, and may also be transferred to thecomputer by wireless means from a download site via an artificialsatellite for digital satellite broadcasting, or may be transferred tothe computer by wired means via a network, such as a LAN (Local AreaNetwork) or the Internet. Then, the computer is able to receive theprogram transferred as described above by a communication unit 108 andinstall the program into the built-in hard disk 105.

The computer has a built-in CPU (Central Processing Unit) 102. Aninput/output interface 110 is connected to the CPU 102 via a bus 101.Upon the input of an instruction by operating an input unit 107, whichconsists of a keyboard, a mouse, or a microphone, by the user via theinput/output interface 110, the CPU 102 executes a program stored in theROM (Read Only Memory) 103. The CPU 102 also loads the following typesof programs into a RAM (Random Access Memory) 104 and executes them: aprogram stored in the hard disk 105, a program transferred from asatellite or a network, received by the communication unit 108, andinstalled into the hard disk 105, and a program read from the removablerecording medium 111 loaded in a drive 109 and installed into the harddisk 105. Accordingly, the CPU 102 executes the processings indicated bythe above-described flowcharts or the processings performed by thefunctions of the above-described block diagrams. The CPU 102 thenoutputs the processing results, if necessary, from an output unit 106,which consists of an LCD (Liquid Crystal Display) or a speaker, ortransmits the processing results from the communication unit 108, orrecords them in the hard disk 105, via the input/output interface 110.

Steps forming the programs for allowing the computer to execute thevarious processings are not necessarily performed in chronological orderas described in the flowcharts of the specification. Alternatively, thesteps may be performed concurrently or individually (for example,concurrent processing or object processing are included).

The programs may be performed by a single computer, or a plurality ofcomputers may be used for performing distribute processing on theprograms. Alternatively, the programs may be transferred to a remotecomputer and be executed.

In this embodiment, the present invention has been described in thecontext of the conversion of SD image data into HD image data. However,the present invention can be applied to the case in which another typeof data, for example, audio data, is converted into higher-quality audiodata.

In the present invention, SD image data may be converted into HD imagedata having a larger number of pixels (number of samples), HD image datahaving improved spatial resolution, HD image data having improvedtemporal resolution (a larger number of frames or fields), or HD imagedata having improved level-direction resolution (a larger number of bitsallocated to the pixel values). Alternatively, the present invention maybe used for enlarging images.

In this embodiment, the image processing apparatus for converting animage and the learning device for learning a tap coefficient for eachclass used in the image processing apparatus are separately formed.However, the image processing apparatus and the learning device may beintegrally formed. In this case, the learning device can performlearning in real time, and the tap coefficients used in the imageprocessing apparatus can be updated in real time.

Although in this embodiment a tap coefficient for each class isprestored in the coefficient RAMs 5, 7, and 77, the tap coefficients maybe supplied to the image processing apparatus together with, forexample, an SD image.

In this embodiment, an HD pixel is determined by a linear equation.However, an HD image may be determined by a quadratic expression or anexpression of a higher degree.

A class tap and a predictive tap may be formed by extracting SD pixelsfrom SD image data of a plurality frames rather than one frame.

Blocks of HD image data may also be formed by HD pixels of HD image dataof a plurality of frames rather than one frame.

The image processing apparatus shown in FIG. 2 or 12 can be applied to,for example, a television receiver for receiving television broadcastsignals and displaying images, a DVD playback apparatus for playing backimage data from DVD and outputting it, or a VTR for playing back imagedata from video tape and outputting it, or an apparatus for processinganother type of image.

Industrial Applicability

As described above, according to the present invention, data can beconverted into higher-quality data.

1. A data conversion apparatus for converting first data into second data, comprising: class-tap generating means for generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; classification means for classifying the specified sample based on the class tap; predictive-tap generating means for generating, from the first data, a predictive tap for determining the specified sample; tap-coefficient obtaining means for obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between supervisor data corresponding to the second data, which serves as a learning supervisor, and learner data corresponding to the first data, which serves as a learner, for each of said at least one class by giving a predetermined constraint condition to the supervisor data; and computation means for determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 2. A data conversion apparatus according to claim 1, wherein the predetermined constraint condition is a condition for imposing constraint on a relationship between a first sample and a second sample of the supervisor-data.
 3. A data conversion apparatus according to claim 2, wherein the predetermined constraint condition is a condition for imposing constraint on a difference between the first sample and the second sample of the supervisor data.
 4. A data conversion apparatus according to claim 3, wherein: the tap coefficient is obtained by learning a relationship between the learner data and the difference of the first sample and the second sample of the supervisor data; and said computation means determines the specified sample by determining the difference between the specified sample and a predetermined sample of the second data other than the specified sample by using the tap coefficient and the predictive tap, and by adding the difference determined for the specified sample to the predetermined sample.
 5. A data conversion apparatus according to claim 3, wherein: the tap coefficient is generated by adding a first tap coefficient obtained by learning the relationship between the learner data and the difference between the first sample and the second sample of the supervisor data to a second tap coefficient obtained by learning the relationship between the learner data and the first sample of the supervisor data; and said computation means determines the specified sample by using the tap coefficient and the predictive tap.
 6. A data conversion apparatus according to claim 1, wherein said computation means determines the specified sample by performing product-sum computation using the tap coefficient and the predictive tap.
 7. A data conversion apparatus according to claim 1, wherein the first data and the second data are image data.
 8. A data conversion apparatus according to claim 1, wherein the second data is higher-quality data than the first data.
 9. A data conversion method for converting first data into second data, comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between supervisor data corresponding to the second data, which serves as a learning supervisor, and learner data corresponding to the first data, which serves as a learner, for each of said at least one class by giving a predetermined constraint condition to the supervisor data; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 10. A program allowing a computer to execute data conversion processing for converting first data into second data, comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between supervisor data corresponding to the second data, which serves as a learning supervisor, and learner data corresponding to the first data, which serves as a learner, for each of said at least one class by giving a predetermined constraint condition to the supervisor data; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 11. A recording medium in which a program allowing a computer to execute data conversion processing for converting first data into second data is recorded, said program comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between supervisor data corresponding to the second data, which serves as a learning supervisor, and learner data corresponding to the first data, which serves as a learner, for each of said at least one class by giving a predetermined constraint condition to the supervisor data; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 12. A learning device for performing learning for determining a predetermined tap coefficient used for converting first data into second data, comprising: class-tap generating means for generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; classification means for classifying the specified item of data based on the class tap; predictive-tap generating means for generating a predictive tap used for determining the specified item of data from the learner data; and learning means for determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between the supervisor data and the learner data for each of said at least one class by giving a predetermined constraint condition to the supervisor data.
 13. A learning device according to claim 12, wherein said learning means learns the relationship between the supervisor data and the learner data by giving a constraint condition for imposing constraint on a relationship between a first sample and a second sample of the supervisor data.
 14. A learning device according to claim 13, wherein said learning means learns the relationship between the supervisor data and the learner data by giving a constraint condition for imposing constraint on a difference between the first sample and the second sample of the supervisor data.
 15. A learning device according to claim 14, wherein said learning means determines the tap coefficient by learning the relationship between the learner data and the difference between the first sample and the second sample of the supervisor data.
 16. A learning device according to claim 14, wherein said learning means determines a first tap coefficient by learning the relationship between the learner data and the difference between the first sample and the second sample of the supervisor data, and also determines a second tap coefficient by learning the relationship between the learner data and the first sample of the supervisor data, and adds the first tap coefficient to the second tap coefficient, thereby determining a final tap coefficient.
 17. A learning device according to claim 12, wherein said learning means determines the tap coefficient for converting the first data into the second data by performing product-sum computation of the first data and the tap coefficient.
 18. A learning device according to claim 12, wherein the first data and the second data are image data.
 19. A learning device according to claim 12, wherein the second data is higher-quality data than the first data.
 20. A learning method for performing learning for determining a predetermined tap coefficient used for converting first data into second data, comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between the supervisor data and the learner data for each of said at least one class by giving a predetermined constraint condition to the supervisor data.
 21. A program allowing a computer to execute learning processing for determining a predetermined tap coefficient used for converting first data into second data, comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between the supervisor data and the learner data for each of said at least one class by giving a predetermined constraint condition to the supervisor data.
 22. A recording medium in which a program allowing a computer to execute learning processing for determining a predetermined tap coefficient used for converting first data into second data is recorded, said program comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between the supervisor data and the learner data for each of said at least one class by giving a predetermined constraint condition to the supervisor data.
 23. A data conversion apparatus for converting first data into second data, comprising: class-tap generating means for generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; classification means for classifying the specified sample based on the class tap; predictive-tap generating means for generating, from the first data, a predictive tap for determining the specified sample; tap-coefficient obtaining means for obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between a feature obtained from a plurality of samples of supervisor data corresponding to the second data, which serves as a learning supervisor, and a plurality of samples of learner data corresponding to the first data, which serves as a learner, for each of said at least one class; and computation means for determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 24. A data conversion apparatus according to claim 23, wherein the plurality of samples of the supervisor data comprise two samples consisting of a first sample and a second sample of the supervisor data.
 25. A data conversion apparatus according to claim 24, wherein the feature obtained from the plurality of samples of the supervisor data is a difference between the first sample and the second sample.
 26. A data conversion apparatus according to claim 25, wherein: the tap coefficient is obtained by learning a relationship between the plurality of samples of the learner data and the difference between the first sample and the second sample of the supervisor data; and said computation means determines the specified sample by determining a difference between the specified sample and a predetermined sample of the second data other than the specified sample by using the tap coefficient and the predictive tap, and by adding the difference determined for the specified sample to the predetermined sample.
 27. A data conversion apparatus according to claim 25, wherein: the tap coefficient is generated by adding a first tap coefficient obtained by learning a relationship between the plurality of samples of the learner data and the difference between the first sample and the second sample of the supervisor data to a second tap coefficient obtained by learning a relationship between the plurality of samples of the learner data and the first sample of the supervisor data; and said computation means determines the specified sample by using the tap coefficient and the predictive tap.
 28. A data conversion apparatus according to claim 23, wherein said computation means determines the specified sample by performing product-sum computation using the tap coefficient and the predictive tap.
 29. A data conversion apparatus according to claim 23, wherein the first data and the second data are image data.
 30. A data conversion apparatus according to claim 23, wherein the second data is higher-quality data than the first data.
 31. A data conversion method for converting first data into second data, comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between a feature obtained from a plurality of samples of supervisor data corresponding to the second data, which serves as a learning supervisor, and a plurality of samples of learner data corresponding to the first data, which serves as a learner, for each of said at least one class; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 32. A program allowing a computer to execute data conversion processing for converting first data into second data, comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between a feature obtained from a plurality of samples of supervisor data corresponding to the second data, which serves as a learning supervisor, and a plurality of samples of learner data corresponding to the first data, which serves as a learner, for each of said at least one class; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 33. A recording medium in which a program allowing a computer to execute data conversion processing for converting first data into second data is recorded, said program comprising: a class-tap generating step of generating, from the first data, a class tap used for classifying a specified sample of the second data into a class of at least one class; a classification step of classifying the specified sample based on the class tap; a predictive-tap generating step of generating, from the first data, a predictive tap for determining the specified sample; a tap-coefficient obtaining step of obtaining a tap coefficient for the class of the specified sample from tap coefficients obtained by learning a relationship between a feature obtained from a plurality of samples of supervisor data corresponding to the second data, which serves as a learning supervisor, and a plurality of samples of learner data corresponding to the first data, which serves as a learner, for each of said at least one class; and a computation step of determining the specified sample by using the predictive tap and the tap coefficient for the class of the specified sample.
 34. A learning device for performing learning for determining a predetermined tap coefficient used for converting first data into second data, comprising: class-tap generating means for generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; classification means for classifying the specified item of data based on the class tap; predictive-tap generating means for generating a predictive tap used for determining the specified item of data from the learner data; and learning means for determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between a feature obtained from a plurality of samples of the supervisor data and a plurality of samples of the learner data for each of said at least one class.
 35. A learning device according to claim 34, wherein said learning means learns a relationship between the plurality of samples of the learner data and two samples consisting of a first sample and a second sample of the supervisor data.
 36. A learning device according to claim 35, wherein said learning means determines a difference between the first sample and the second sample of the supervisor data as the feature obtained from the plurality of samples of the supervisor data.
 37. A learning device according to claim 36, wherein said learning means determines the tap coefficient by learning a relationship between the plurality of samples of the learner data and the difference between the first sample and the second sample of the supervisor data.
 38. A learning device according to claim 36, wherein said learning means determines a first tap coefficient by learning the relationship between the plurality of samples of the learner data and the difference between the first sample and the second sample of the supervisor data, and also determines a second tap coefficient by learning a relationship between the plurality of samples of the learner data and the first sample of the supervisor data, and adds the first tap coefficient to the second tap coefficient, thereby determining a final tap coefficient.
 39. A learning device according to claim 34, wherein said learning means determines the tap coefficient for converting the first data into the second data by performing product-sum computation of the first data and the tap coefficient.
 40. A learning device according to claim 34, wherein the first data and the second data are image data.
 41. A learning device according to claim 34, wherein the second data is higher-quality data than the first data.
 42. A learning method for performing learning for determining a predetermined tap coefficient used for converting first data into second data, comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between a feature obtained from a plurality of samples of the supervisor data and a plurality of samples of the learner data for each of said at least one class.
 43. A program allowing a computer to execute learning processing for determining a predetermined tap coefficient used for converting first data into second data, comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between a feature obtained from a plurality of samples of the supervisor data and a plurality of samples of the learner data for each of said at least one class.
 44. A recording medium in which a program allowing a computer to execute learning processing for determining a predetermined tap coefficient used for converting first data into second data is recorded, said program comprising: a class-tap generating step of generating a class tap used for classifying a specified item of supervisor data corresponding to the second data, which serves as a supervisor for learning the tap coefficient, into a class of at least one class, from learner data corresponding to the first data, which serves as a learner; a classification step of classifying the specified item of data based on the class tap; a predictive-tap generating step of generating a predictive tap used for determining the specified item of data from the learner data; and a learning step of determining, by using the specified item of data and the predictive tap, the tap coefficient for each of said at least one class by learning a relationship between a feature obtained from a plurality of samples of the supervisor data and a plurality of samples of the learner data for each of said at least one class. 